Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

_{Comment @coderabbitai help to get the list of available commands and usage tips.}

Thanks for the contribution, Zhuo! The feature itself is small and the tests are nice — but the diff feels heavier than it needs to be because backend-type branching is leaking into orchestration code. A few things to consider before merge:

1. is_mlx() branching is repeated 4× — push it into the wrapper

The same if complete.is_mlx() { resolve_mlx_matched_stop_json(...) } else { complete.matched_stop_json() } block appears in:

• model_gateway/src/routers/grpc/regular/processor.rs:187 (chat)
• model_gateway/src/routers/grpc/regular/processor.rs:834 (completion)
• model_gateway/src/routers/grpc/regular/streaming.rs:495 (chat stream)
• model_gateway/src/routers/grpc/regular/streaming.rs:2589 (completion stream)

proto_wrapper.rs already encapsulates backend differences (matched_stop_json(), is_mlx_streaming_finished(), etc.). Could you add a matched_stop_json_with_context(stop, stop_token_ids, tokenizer) method on ProtoGenerateComplete that internally dispatches on the variant? Each caller collapses to one line, and mlx_matched_stop_token_id() no longer needs to be pub. Composes better when the next backend hits the same quirk.

2. apply_mlx_stop_sequences is double-gated

model_gateway/src/routers/grpc/common/stages/helpers.rs:82 pattern-matches ProtoGenerateRequest::Mlx(req), but both callers (chat/request_building.rs:111, completion/request_building.rs:90) already gate on builder_client.is_mlx(). Pick one — either the helper is unconditional and no-ops on non-MLX, or callers gate and the helper assumes MLX.

3. Silent encode-error swallow is a footgun

chat_utils.rs:443 — stop_strings_to_token_ids correctly returns Err (→ 400) for multi-token strings, but silently warn!s and drops when the tokenizer errors or returns zero tokens. The new MockTokenizer::failing() test at line 904 locks this in:

// "Tokenizer errors are silently skipped; the function returns Ok(empty)."
assert!(result.unwrap().is_empty());

Both are the same class of "we cannot honor your stop condition" failure — caller sets stop: ["foo"], encoder errors, request proceeds to max_tokens, user blames SMG. Recommend surfacing both as 400.

4. Is test-case = "3.3.1" worth a new workspace dep?

It's introduced as a brand-new workspace dependency just for ~13 parameterized cases across 2 functions in chat_utils.rs. Not used anywhere else in the workspace today. A plain #[test] with a slice of (input, expected, name) tuples and a loop gets the same coverage without the dep — and keeps the workspace lean. Up to you, but worth weighing.

Smaller items

• streaming.rs got 6 new matched_stop: None struct-literal additions (lines 2422, 2450, 2470, 2505, 2529, 2552). Not blocking, but a ..Default::default() or a constructor on CompletionStreamChoice would prevent this sprawl next time.
• Could you run the curl from the PR description on your local Mac (SMG + MLX backend) and paste the actual response in the test plan? Stream + non-stream, single stop string + multi-token rejection. That's the strongest signal we'll get for a Mac-specific path.
• mlx_engine.rs:245 comment update is good; retaining reject_stop_strings for messages/generate paths is correct.

After (1)–(3), the MLX surface area shrinks a lot and the "MLX branches everywhere" feel goes away.

Hi @zach-li-sudo, the DCO sign-off check has failed. All commits must include a Signed-off-by line.

To fix existing commits:

# Sign off the last N commits (replace N with the number of unsigned commits)
git rebase HEAD~N --signoff
git push --force-with-lease

To sign off future commits automatically:

• Use git commit -s every time, or
• VSCode: enable Git: Always Sign Off in Settings
• PyCharm: enable Sign-off commit in the Commit tool window

Manual testing results: stop sequence with MLX backend

1. SMG with MLX backend setup

1. cargo build
2. Install the gRPC servicer (first time only):

pip install -e grpc_servicer/

3. Start the MLX backend:

source .venv/bin/activate && python -m smg_grpc_servicer.mlx.server \
  --model mlx-community/Qwen3-4B-Instruct-2507-4bit --port 50051

4. Start the SMG gateway server:

./target/debug/smg --worker-urls grpc://localhost:50051 --port 3000

2. Testing scenarios

This PR is to support string stop array in regular
paths: chat, chat completions, streaming chat, streaming chat completion.

Key differences from vLLM

The gateway converts single-token stop strings to token IDs for MLX requests (the MLX proto has
no native string stop field). Multi-token stop strings are rejected with HTTP 400
(unsupported_stop_string), since MLX can only stop on a single token at a time.

Token ID reference

Token IDs used in the tests below assume the Qwen3 tokenizer, which shares vocabulary with Qwen2.5:

• Token 20 → "5"
• Token 21 → "6"

Verify with the tokenizer if results are unexpected:

from mlx_lm import load
_, tokenizer = load("mlx-community/Qwen3-4B-Instruct-2507-4bit")
print(tokenizer.encode("5 6"))   # check IDs for context-free "5" and "6"

3. Results

Test matrix: 4 paths × 5 stop modes = 20 cases.

Single-token stop string cases (3.x.1): PASS = 200, matched_stop = the stop string.
Multi-token stop string cases (3.x.2): PASS = 400 with an appropriate error message.
stop_token_ids cases (3.x.3): PASS = 200, matched_stop = the matched token ID integer.
Combined single-token stop string + stop_token_ids (3.x.4): PASS = 200, matched_stop = integer or string depending on which stop fires first.
Combined multi-token stop string + stop_token_ids (3.x.5): PASS = 400 — multi-token string is rejected regardless of stop_token_ids.

Reading the streaming output below: Only the first content chunk and the final (stop) chunk are
shown. Intermediate content chunks are collapsed to # ... (content: "...") ....

3.1 Chat, non-stream

3.1.1 — stop string, single token ("stop": ["6"]) — expect HTTP 200

curl http://localhost:3000/v1/chat/completions -s -w "\nHTTP %{http_code}" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "mlx-community/Qwen3-4B-Instruct-2507-4bit",
    "messages": [{"role": "user", "content": "Count from 1 to 10, one number per line"}],
    "stop": ["6"],
    "stream": false,
    "max_tokens": 100
  }' | jq -Rs 'split("\n") | {response: .[0] | fromjson, status: .[-1]}'

{
  "response": {
    "id": "chatcmpl-019e12f3-af39-7bc1-a46c-4173bebe0cbc",
    "object": "chat.completion",
    "created": 1778434420,
    "model": "mlx-community/Qwen3-4B-Instruct-2507-4bit",
    "choices": [
      {
        "index": 0,
        "message": {
          "role": "assistant",
          "content": "1  \n2  \n3  \n4  \n5  \n",
          "reasoning_content": null
        },
        "finish_reason": "stop",
        "matched_stop": "6"
      }
    ],
    "usage": {
      "prompt_tokens": 21,
      "completion_tokens": 11,
      "total_tokens": 32
    },
    "system_fingerprint": "default"
  },
  "status": "HTTP 200"
}

3.1.2 — stop string, multi-token ("stop": ["hello world"]) — expect HTTP 400

curl http://localhost:3000/v1/chat/completions -s -w "\nHTTP %{http_code}" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "mlx-community/Qwen3-4B-Instruct-2507-4bit",
    "messages": [{"role": "user", "content": "Say: hi there and hello world!"}],
    "stop": ["hello world"],
    "stream": false,
    "max_tokens": 100
  }' | jq -Rs 'split("\n") | {response: .[0] | fromjson, status: .[-1]}'

{
  "response": {
    "error": {
      "type": "Bad Request",
      "code": "unsupported_stop_string",
      "message": "stop string \"hello world\" encodes to 2 tokens; MLX backend only supports single-token stop strings",
      "param": null
    }
  },
  "status": "HTTP 400"
}

3.1.3 — stop_token_ids ([20, 21] = tokens for "5" and "6")

curl http://localhost:3000/v1/chat/completions -s \
  -H "Content-Type: application/json" \
  -d '{
    "model": "mlx-community/Qwen3-4B-Instruct-2507-4bit",
    "messages": [{"role": "user", "content": "Count from 1 to 10, one number per line"}],
    "stop_token_ids": [20, 21],
    "stream": false,
    "max_tokens": 100
  }' | jq

{
  "id": "chatcmpl-019e12f3-b06a-7e30-9026-b277ef4cd022",
  "object": "chat.completion",
  "created": 1778434420,
  "model": "mlx-community/Qwen3-4B-Instruct-2507-4bit",
  "choices": [
    {
      "index": 0,
      "message": {
        "role": "assistant",
        "content": "1  \n2  \n3  \n4  \n",
        "reasoning_content": null
      },
      "finish_reason": "stop",
      "matched_stop": 20
    }
  ],
  "usage": {
    "prompt_tokens": 21,
    "completion_tokens": 9,
    "total_tokens": 30
  },
  "system_fingerprint": "default"
}

3.1.4 — combined: single-token stop string + stop_token_ids ("stop": ["6"], "stop_token_ids": [20]) — expect HTTP 200

curl http://localhost:3000/v1/chat/completions -s -w "\nHTTP %{http_code}" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "mlx-community/Qwen3-4B-Instruct-2507-4bit",
    "messages": [{"role": "user", "content": "Count from 1 to 10, one number per line"}],
    "stop": ["6"],
    "stop_token_ids": [20],
    "stream": false,
    "max_tokens": 100
  }' | jq -Rs 'split("\n") | {response: .[0] | fromjson, status: .[-1]}'

{
  "response": {
    "id": "chatcmpl-019e13c9-0ea1-77c0-b430-bf53fbcdede2",
    "object": "chat.completion",
    "created": 1778448404,
    "model": "mlx-community/Qwen3-4B-Instruct-2507-4bit",
    "choices": [
      {
        "index": 0,
        "message": {
          "role": "assistant",
          "content": "1  \n2  \n3  \n4  \n",
          "reasoning_content": null
        },
        "finish_reason": "stop",
        "matched_stop": 20
      }
    ],
    "usage": {
      "prompt_tokens": 21,
      "completion_tokens": 9,
      "total_tokens": 30
    },
    "system_fingerprint": "default"
  },
  "status": "HTTP 200"
}

3.1.5 — combined: multi-token stop string + stop_token_ids ("stop": ["hello world"], "stop_token_ids": [20, 21]) — expect HTTP 400

curl http://localhost:3000/v1/chat/completions -s -w "\nHTTP %{http_code}" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "mlx-community/Qwen3-4B-Instruct-2507-4bit",
    "messages": [{"role": "user", "content": "Count from 1 to 10, one number per line"}],
    "stop": ["hello world"],
    "stop_token_ids": [20, 21],
    "stream": false,
    "max_tokens": 100
  }' | jq -Rs 'split("\n") | {response: .[0] | fromjson, status: .[-1]}'

{
  "response": {
    "error": {
      "type": "Bad Request",
      "code": "unsupported_stop_string",
      "message": "stop string \"hello world\" encodes to 2 tokens; MLX backend only supports single-token stop strings",
      "param": null
    }
  },
  "status": "HTTP 400"
}

3.2 Chat, stream

3.2.1 — stop string, single token ("stop": ["6"]) — expect HTTP 200

curl http://localhost:3000/v1/chat/completions -s -N -w "\nHTTP %{http_code}" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "mlx-community/Qwen3-4B-Instruct-2507-4bit",
    "messages": [{"role": "user", "content": "Count from 1 to 10, one number per line"}],
    "stop": ["6"],
    "stream": true,
    "max_tokens": 100
  }'

data: {"id":"chatcmpl-019e12f4-6c1f-7651-a730-137fffaab34b","object":"chat.completion.chunk","created":1778434468,"model":"mlx-community/Qwen3-4B-Instruct-2507-4bit","system_fingerprint":"default","choices":[{"index":0,"delta":{"role":"assistant","reasoning_content":null},"logprobs":null,"finish_reason":null}]}

data: {"id":"chatcmpl-019e12f4-6c1f-7651-a730-137fffaab34b","object":"chat.completion.chunk","created":1778434468,...,"choices":[{"index":0,"delta":{"role":"assistant","content":"1  ","reasoning_content":null},"logprobs":null,"finish_reason":null}]}

# ... (content: "1  \n2  \n3  \n4  \n5  \n") ...

data: {"id":"chatcmpl-019e12f4-6c1f-7651-a730-137fffaab34b","object":"chat.completion.chunk","created":1778434468,"model":"mlx-community/Qwen3-4B-Instruct-2507-4bit","system_fingerprint":"default","choices":[{"index":0,"delta":{"reasoning_content":null},"logprobs":null,"finish_reason":"stop","matched_stop":"6"}]}

data: [DONE]

HTTP 200

3.2.2 — stop string, multi-token ("stop": ["hello world"]) — expect HTTP 400

curl http://localhost:3000/v1/chat/completions -s -N -w "\nHTTP %{http_code}" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "mlx-community/Qwen3-4B-Instruct-2507-4bit",
    "messages": [{"role": "user", "content": "Repeat exactly: 1 2 3 hello world 4 5"}],
    "stop": ["hello world"],
    "stream": true,
    "max_tokens": 100
  }'

{"error":{"type":"Bad Request","code":"unsupported_stop_string","message":"stop string \"hello world\" encodes to 2 tokens; MLX backend only supports single-token stop strings","param":null}}
HTTP 400

3.2.3 — stop_token_ids ([20, 21] = tokens for "5" and "6")

curl http://localhost:3000/v1/chat/completions -s -N \
  -H "Content-Type: application/json" \
  -d '{
    "model": "mlx-community/Qwen3-4B-Instruct-2507-4bit",
    "messages": [{"role": "user", "content": "Count from 1 to 10, one number per line"}],
    "stop_token_ids": [20, 21],
    "stream": true,
    "max_tokens": 100
  }'

data: {"id":"chatcmpl-019e12f4-6d47-7d00-beca-1339545bf2bd","object":"chat.completion.chunk","created":1778434469,"model":"mlx-community/Qwen3-4B-Instruct-2507-4bit","system_fingerprint":"default","choices":[{"index":0,"delta":{"role":"assistant","reasoning_content":null},"logprobs":null,"finish_reason":null}]}

data: {"id":"chatcmpl-019e12f4-6d47-7d00-beca-1339545bf2bd","object":"chat.completion.chunk","created":1778434469,...,"choices":[{"index":0,"delta":{"role":"assistant","content":"1","reasoning_content":null},"logprobs":null,"finish_reason":null}]}

# ... (content: "1  \n2  \n3  \n4  \n") ...

data: {"id":"chatcmpl-019e12f4-6d47-7d00-beca-1339545bf2bd","object":"chat.completion.chunk","created":1778434469,"model":"mlx-community/Qwen3-4B-Instruct-2507-4bit","system_fingerprint":"default","choices":[{"index":0,"delta":{"reasoning_content":null},"logprobs":null,"finish_reason":"stop","matched_stop":20}]}

data: [DONE]

3.2.4 — combined: single-token stop string + stop_token_ids ("stop": ["6"], "stop_token_ids": [20]) — expect HTTP 200

curl http://localhost:3000/v1/chat/completions -s -N -w "\nHTTP %{http_code}" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "mlx-community/Qwen3-4B-Instruct-2507-4bit",
    "messages": [{"role": "user", "content": "Count from 1 to 10, one number per line"}],
    "stop": ["6"],
    "stop_token_ids": [20],
    "stream": true,
    "max_tokens": 100
  }'

data: {"id":"chatcmpl-019e13c9-3610-7a81-928c-7aa1e5f75f9d","object":"chat.completion.chunk","created":1778448414,"model":"mlx-community/Qwen3-4B-Instruct-2507-4bit","system_fingerprint":"default","choices":[{"index":0,"delta":{"role":"assistant","reasoning_content":null},"logprobs":null,"finish_reason":null}]}

data: {"id":"chatcmpl-019e13c9-3610-7a81-928c-7aa1e5f75f9d","object":"chat.completion.chunk","created":1778448414,...,"choices":[{"index":0,"delta":{"role":"assistant","content":"1  ","reasoning_content":null},"logprobs":null,"finish_reason":null}]}

# ... (content: "1  \n2  \n3  \n4  \n") ...

data: {"id":"chatcmpl-019e13c9-3610-7a81-928c-7aa1e5f75f9d","object":"chat.completion.chunk","created":1778448414,"model":"mlx-community/Qwen3-4B-Instruct-2507-4bit","system_fingerprint":"default","choices":[{"index":0,"delta":{"reasoning_content":null},"logprobs":null,"finish_reason":"stop","matched_stop":20}]}

data: [DONE]

HTTP 200

3.2.5 — combined: multi-token stop string + stop_token_ids ("stop": ["hello world"], "stop_token_ids": [20, 21]) — expect HTTP 400

curl http://localhost:3000/v1/chat/completions -s -N -w "\nHTTP %{http_code}" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "mlx-community/Qwen3-4B-Instruct-2507-4bit",
    "messages": [{"role": "user", "content": "Count from 1 to 10, one number per line"}],
    "stop": ["hello world"],
    "stop_token_ids": [20, 21],
    "stream": true,
    "max_tokens": 100
  }'

{"error":{"type":"Bad Request","code":"unsupported_stop_string","message":"stop string \"hello world\" encodes to 2 tokens; MLX backend only supports single-token stop strings","param":null}}
HTTP 400

3.3 Completion, non-stream

3.3.1 — stop string, single token ("stop": ["6"]) — expect HTTP 200

curl http://localhost:3000/v1/completions -s -w "\nHTTP %{http_code}" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "mlx-community/Qwen3-4B-Instruct-2507-4bit",
    "prompt": "Count from 1 to 10, one number per line:\n1\n2\n3\n4\n",
    "stop": ["6"],
    "stream": false,
    "max_tokens": 100
  }' | jq -Rs 'split("\n") | {response: .[0] | fromjson, status: .[-1]}'

{
  "response": {
    "id": "cmpl_019e12f4-9eda-71c2-bf57-ab5fca3036f6",
    "object": "text_completion",
    "created": 1778434481,
    "model": "mlx-community/Qwen3-4B-Instruct-2507-4bit",
    "choices": [
      {
        "text": "5\n",
        "index": 0,
        "finish_reason": "stop",
        "matched_stop": "6"
      }
    ],
    "usage": {
      "prompt_tokens": 22,
      "completion_tokens": 3,
      "total_tokens": 25
    },
    "system_fingerprint": "default"
  },
  "status": "HTTP 200"
}

3.3.2 — stop string, multi-token ("stop": ["hello world"]) — expect HTTP 400

curl http://localhost:3000/v1/completions -s -w "\nHTTP %{http_code}" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "mlx-community/Qwen3-4B-Instruct-2507-4bit",
    "prompt": "Repeat exactly: 1 2 3 hello world 4 5",
    "stop": ["hello world"],
    "stream": false,
    "max_tokens": 100
  }' | jq -Rs 'split("\n") | {response: .[0] | fromjson, status: .[-1]}'

{
  "response": {
    "error": {
      "type": "Bad Request",
      "code": "unsupported_stop_string",
      "message": "stop string \"hello world\" encodes to 2 tokens; MLX backend only supports single-token stop strings",
      "param": null
    }
  },
  "status": "HTTP 400"
}

3.3.3 — stop_token_ids ([20, 21] = tokens for "5" and "6")

curl http://localhost:3000/v1/completions -s \
  -H "Content-Type: application/json" \
  -d '{
    "model": "mlx-community/Qwen3-4B-Instruct-2507-4bit",
    "prompt": "Count from 1 to 10, one number per line:\n1\n2\n3\n4\n",
    "stop_token_ids": [20, 21],
    "stream": false,
    "max_tokens": 100
  }' | jq

{
  "id": "cmpl_019e12f4-9faf-7582-bcc0-bb7849cf6585",
  "object": "text_completion",
  "created": 1778434482,
  "model": "mlx-community/Qwen3-4B-Instruct-2507-4bit",
  "choices": [
    {
      "text": "",
      "index": 0,
      "finish_reason": "stop",
      "matched_stop": 20
    }
  ],
  "usage": {
    "prompt_tokens": 22,
    "completion_tokens": 1,
    "total_tokens": 23
  },
  "system_fingerprint": "default"
}

Empty text because the stop token was the very next token after the prompt — stopped immediately.

3.3.4 — combined: single-token stop string + stop_token_ids ("stop": ["6"], "stop_token_ids": [20]) — expect HTTP 200

curl http://localhost:3000/v1/completions -s -w "\nHTTP %{http_code}" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "mlx-community/Qwen3-4B-Instruct-2507-4bit",
    "prompt": "Count from 1 to 10, one number per line:\n1\n2\n3\n4\n",
    "stop": ["6"],
    "stop_token_ids": [20],
    "stream": false,
    "max_tokens": 100
  }' | jq -Rs 'split("\n") | {response: .[0] | fromjson, status: .[-1]}'

{
  "response": {
    "id": "cmpl_019e13c9-6957-7f10-8038-53f7c215dd87",
    "object": "text_completion",
    "created": 1778448427,
    "model": "mlx-community/Qwen3-4B-Instruct-2507-4bit",
    "choices": [
      {
        "text": "",
        "index": 0,
        "finish_reason": "stop",
        "matched_stop": 20
      }
    ],
    "usage": {
      "prompt_tokens": 22,
      "completion_tokens": 1,
      "total_tokens": 23
    },
    "system_fingerprint": "default"
  },
  "status": "HTTP 200"
}

3.3.5 — combined: multi-token stop string + stop_token_ids ("stop": ["hello world"], "stop_token_ids": [20, 21]) — expect HTTP 400

curl http://localhost:3000/v1/completions -s -w "\nHTTP %{http_code}" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "mlx-community/Qwen3-4B-Instruct-2507-4bit",
    "prompt": "Count from 1 to 10, one number per line:\n1\n2\n3\n4\n",
    "stop": ["hello world"],
    "stop_token_ids": [20, 21],
    "stream": false,
    "max_tokens": 100
  }' | jq -Rs 'split("\n") | {response: .[0] | fromjson, status: .[-1]}'

{
  "response": {
    "error": {
      "type": "Bad Request",
      "code": "unsupported_stop_string",
      "message": "stop string \"hello world\" encodes to 2 tokens; MLX backend only supports single-token stop strings",
      "param": null
    }
  },
  "status": "HTTP 400"
}

3.4 Completion, stream

3.4.1 — stop string, single token ("stop": ["6"]) — expect HTTP 200

curl http://localhost:3000/v1/completions -s -N -w "\nHTTP %{http_code}" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "mlx-community/Qwen3-4B-Instruct-2507-4bit",
    "prompt": "Count from 1 to 10, one number per line:\n1\n2\n3\n4\n",
    "stop": ["6"],
    "stream": true,
    "max_tokens": 100
  }'

data: {"id":"cmpl_019e12f4-bb3b-7b80-b016-13a6be12605c","object":"text_completion","created":1778434489,"choices":[{"text":"5","index":0,"finish_reason":null}],"model":"mlx-community/Qwen3-4B-Instruct-2507-4bit","system_fingerprint":"default"}

# ... (content: "5\n") ...

data: {"id":"cmpl_019e12f4-bb3b-7b80-b016-13a6be12605c","object":"text_completion","created":1778434489,"choices":[{"text":"","index":0,"finish_reason":"stop"}],"model":"mlx-community/Qwen3-4B-Instruct-2507-4bit","system_fingerprint":"default"}

data: [DONE]

HTTP 200

3.4.2 — stop string, multi-token ("stop": ["hello world"]) — expect HTTP 400

curl http://localhost:3000/v1/completions -s -N -w "\nHTTP %{http_code}" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "mlx-community/Qwen3-4B-Instruct-2507-4bit",
    "prompt": "Repeat exactly: 1 2 3 hello world 4 5",
    "stop": ["hello world"],
    "stream": true,
    "max_tokens": 100
  }'

{"error":{"type":"Bad Request","code":"unsupported_stop_string","message":"stop string \"hello world\" encodes to 2 tokens; MLX backend only supports single-token stop strings","param":null}}
HTTP 400

3.4.3 — stop_token_ids ([20, 21] = tokens for "5" and "6")

curl http://localhost:3000/v1/completions -s -N \
  -H "Content-Type: application/json" \
  -d '{
    "model": "mlx-community/Qwen3-4B-Instruct-2507-4bit",
    "prompt": "Count from 1 to 10, one number per line:\n1\n2\n3\n4\n",
    "stop_token_ids": [20, 21],
    "stream": true,
    "max_tokens": 100
  }'

data: {"id":"cmpl_019e12f4-bc18-7b71-b9a6-fb673df80244","object":"text_completion","created":1778434489,"choices":[{"text":"","index":0,"finish_reason":"stop"}],"model":"mlx-community/Qwen3-4B-Instruct-2507-4bit","system_fingerprint":"default"}

data: [DONE]

3.4.4 — combined: single-token stop string + stop_token_ids ("stop": ["6"], "stop_token_ids": [20]) — expect HTTP 200

curl http://localhost:3000/v1/completions -s -N -w "\nHTTP %{http_code}" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "mlx-community/Qwen3-4B-Instruct-2507-4bit",
    "prompt": "Count from 1 to 10, one number per line:\n1\n2\n3\n4\n",
    "stop": ["6"],
    "stop_token_ids": [20],
    "stream": true,
    "max_tokens": 100
  }'

data: {"id":"cmpl_019e13c9-8d8d-75a0-bba2-d7da2fc33066","object":"text_completion","created":1778448436,"choices":[{"text":"","index":0,"finish_reason":"stop"}],"model":"mlx-community/Qwen3-4B-Instruct-2507-4bit","system_fingerprint":"default"}

data: [DONE]

HTTP 200

3.4.5 — combined: multi-token stop string + stop_token_ids ("stop": ["hello world"], "stop_token_ids": [20, 21]) — expect HTTP 400

curl http://localhost:3000/v1/completions -s -N -w "\nHTTP %{http_code}" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "mlx-community/Qwen3-4B-Instruct-2507-4bit",
    "prompt": "Count from 1 to 10, one number per line:\n1\n2\n3\n4\n",
    "stop": ["hello world"],
    "stop_token_ids": [20, 21],
    "stream": true,
    "max_tokens": 100
  }'

{"error":{"type":"Bad Request","code":"unsupported_stop_string","message":"stop string \"hello world\" encodes to 2 tokens; MLX backend only supports single-token stop strings","param":null}}
HTTP 400

Thanks for the contribution, Zhuo! The feature itself is small and the tests are nice — but the diff feels heavier than it needs to be because backend-type branching is leaking into orchestration code. A few things to consider before merge:

1. is_mlx() branching is repeated 4× — push it into the wrapper

The same if complete.is_mlx() { resolve_mlx_matched_stop_json(...) } else { complete.matched_stop_json() } block appears in:

• model_gateway/src/routers/grpc/regular/processor.rs:187 (chat)
• model_gateway/src/routers/grpc/regular/processor.rs:834 (completion)
• model_gateway/src/routers/grpc/regular/streaming.rs:495 (chat stream)
• model_gateway/src/routers/grpc/regular/streaming.rs:2589 (completion stream)

proto_wrapper.rs already encapsulates backend differences (matched_stop_json(), is_mlx_streaming_finished(), etc.). Could you add a matched_stop_json_with_context(stop, stop_token_ids, tokenizer) method on ProtoGenerateComplete that internally dispatches on the variant? Each caller collapses to one line, and mlx_matched_stop_token_id() no longer needs to be pub. Composes better when the next backend hits the same quirk.

2. apply_mlx_stop_sequences is double-gated

model_gateway/src/routers/grpc/common/stages/helpers.rs:82 pattern-matches ProtoGenerateRequest::Mlx(req), but both callers (chat/request_building.rs:111, completion/request_building.rs:90) already gate on builder_client.is_mlx(). Pick one — either the helper is unconditional and no-ops on non-MLX, or callers gate and the helper assumes MLX.

3. Silent encode-error swallow is a footgun

chat_utils.rs:443 — stop_strings_to_token_ids correctly returns Err (→ 400) for multi-token strings, but silently warn!s and drops when the tokenizer errors or returns zero tokens. The new MockTokenizer::failing() test at line 904 locks this in:

// "Tokenizer errors are silently skipped; the function returns Ok(empty)."
assert!(result.unwrap().is_empty());

Both are the same class of "we cannot honor your stop condition" failure — caller sets stop: ["foo"], encoder errors, request proceeds to max_tokens, user blames SMG. Recommend surfacing both as 400.

4. Is test-case = "3.3.1" worth a new workspace dep?

It's introduced as a brand-new workspace dependency just for ~13 parameterized cases across 2 functions in chat_utils.rs. Not used anywhere else in the workspace today. A plain #[test] with a slice of (input, expected, name) tuples and a loop gets the same coverage without the dep — and keeps the workspace lean. Up to you, but worth weighing.

Smaller items

• streaming.rs got 6 new matched_stop: None struct-literal additions (lines 2422, 2450, 2470, 2505, 2529, 2552). Not blocking, but a ..Default::default() or a constructor on CompletionStreamChoice would prevent this sprawl next time.
• Could you run the curl from the PR description on your local Mac (SMG + MLX backend) and paste the actual response in the test plan? Stream + non-stream, single stop string + multi-token rejection. That's the strongest signal we'll get for a Mac-specific path.
• mlx_engine.rs:245 comment update is good; retaining reject_stop_strings for messages/generate paths is correct.

After (1)–(3), the MLX surface area shrinks a lot and the "MLX branches everywhere" feel goes away.

Hi Keyang, thanks for your comments! The issues you've mentioned are resolved as follows:

1. is_mlx() branching is repeated 4× — push it into the wrapper: fixed in commit #6f390899 by moving mlx match_stop processing logic into proto wrapper
2. apply_mlx_stop_sequences is double-gated: fixed in commit #15c12271 fix double gated apply_mlx_stop_sequences: helper is unconditional and no-ops on non-MLX
3. Silent encode-error swallow is a footgun: fixed in commit #271cec93 fix silent encode-error: zero token and failed tokenizer throw 400 error
4. Is test-case = "3.3.1" worth a new workspace dep: dev dependency removed, newly added unit tests are refactored using a plain #[test] with a slice of (input, expected, name) tuples and a loop.

Other items:

• streaming.rs got 6 new matched_stop: None struct-literal additions (lines 2422, 2450, 2470, 2505, 2529, 2552). Not blocking, but a ..Default::default() or a constructor on CompletionStreamChoice would prevent this sprawl next time.

Solution: used ..Default::default() fror the construct CompletionStreamChoice fields. See #94eef5a2.

• Could you run the curl from the PR description on your local Mac (SMG + MLX backend) and paste the actual response in the test plan? Stream + non-stream, single stop string + multi-token rejection. That's the strongest signal we'll get for a Mac-specific path.

Yes, the overall test plan executed on my local Mac with real response are done! All test cases are summarized in this test matrix, including {stream, non-stream} x {chat, chat completion} x {stop token ids, stop strings, combined, ...}. Please check linked comment.

I see some other gaps (not directly correlated to this PR) in MLX backend support, like harmony paths, messages/generate endpoints etc. Will discuss these with my investigation later.